Skip to content

feat(hub): whole-package zip download + file list on the agent page - #1843

Merged
kovtcharov-amd merged 2 commits into
mainfrom
feat/hub-package-zip-download
Jun 24, 2026
Merged

feat(hub): whole-package zip download + file list on the agent page#1843
kovtcharov-amd merged 2 commits into
mainfrom
feat/hub-package-zip-download

Conversation

@kovtcharov

Copy link
Copy Markdown
Contributor

Why this matters

The hub agent page showed only metadata — a visitor couldn't actually grab the package, and any "download" would have been a single platform binary, not the whole thing. This adds a one-click Download .zip (one archive containing the npm client, all docs, and every platform binary) plus a file list of exactly what's inside — so anyone can pull the complete agent for any OS straight from hub.amd-gaia.ai/hub/<id>.

Design (chosen up front): one cross-platform zip, pre-built at release time (the publish pipeline assembles + uploads it; the hub just links to a stored object).

What's in it

  • Release pipeline (release_agent_email.yml): after the binaries publish and the lock is regenerated with real hashes, assemble agent-email-<version>.zipbinaries/ (all platforms) + dist/ (npm client) + README/SPEC/SKILL/CHANGELOG + binaries.lock.json + gaia-agent.yaml + LICENSE — emit a package-files.json listing (new gen_package_files.py), and POST both (zip artifact + package_files).
  • Worker: optional package_files part on /publish → stored per-version → paired with the .zip artifact into the catalog's package ({ filename, size_bytes, files }). storage/publish/catalog/types/schema + Worker README + 3 tests.
  • Website: a Download .zip button (with size) and a collapsible file list on the agent page, hidden gracefully when no package was published.

Test plan

  • Worker: cd workers/agent-hub && npm test (63 pass) + npx tsc --noEmit
  • Website: cd website && npx astro check (0 errors) + npx vitest run (19) + build
  • Helper: python hub/agents/python/email/packaging/gen_package_files.py <dir> out.json → sorted {files:[{name,size_bytes}]}
  • publish_to_r2.py --help shows --package-files
  • After a 0.2.1 release on the new pipeline: /hub/email shows the Download .zip button + file list (the zip appears live only once a release publishes it — pre-built at release time)

Before: the hub agent page showed only metadata — no way to grab the actual
package. After: a one-click 'Download .zip' (one archive with the client, docs,
and ALL platform binaries) plus a collapsible file list of what's inside, so
anyone can pull the complete agent for any OS from hub.amd-gaia.ai/hub/<id>.
Pre-built at release time.

- Release: release_agent_email.yml assembles agent-email-<version>.zip (all
  platform binaries + npm dist/ + README/SPEC/SKILL/CHANGELOG + binaries.lock.json
  + gaia-agent.yaml + LICENSE) after the lock is regenerated, plus a
  package-files.json listing (new gen_package_files.py helper), and publishes both.
- Worker: optional 'package_files' part on /publish -> stored per-version -> the
  catalog's package { filename, size_bytes, files } (paired with the .zip
  artifact). storage/publish/catalog/types/schema + Worker README + 3 tests.
- Website: a Download .zip button + collapsible file list on the agent page,
  hidden gracefully when no package was published.

Worker 63 tests + tsc clean; website 0 errors + 19 tests; helper smoke-tested.
@github-actions github-actions Bot added devops DevOps/infrastructure changes website GAIA website (amd-gaia.ai) labels Jun 24, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Verdict: Request changes — one blocking bug means the feature silently never appears in production, even though every test passes.

This PR adds a one-click "Download .zip" (all platform binaries + npm client + docs in a single archive) plus a file list to the hub agent page, assembled and uploaded at release time. The design is clean and the Worker/website/schema/docs are all kept in sync.

The blocker: the release pipeline publishes the platform binaries first (which creates the version), then publishes the zip + file list in a second POST to that same version. But the Worker only stores the file-list part on the first publish of a version — so on the real release path the file list is silently dropped, and the "Download .zip" button + file list never render. The tests publish the zip and file list together in a single call against a fresh state, so they pass while the production two-step flow fails. This is exactly the "test from the user's real state, not your primed one" trap from CLAUDE.md. Fix: store the package file list whenever the part is present (not gated on first-publish), and add a test that publishes binaries first, then the zip second, and asserts the package still surfaces.

🔍 Technical details

🔴 Critical

package_files is dropped on the real release path → feature never renders (workers/agent-hub/src/publish.ts:262)

package_files is stored only inside the if (!versionExists) block:

if (!versionExists) {
  ...
  if (packageFilesText != null) {
    await env.BUCKET.put(packageFilesKey(...), packageFilesText, {...});
  }
}

versionExists = Boolean(existing?.versions[manifest.version]) (publish.ts:200). In the release workflow the version is created by the binaries publish step (release_agent_email.yml:457, same ${MANIFEST} → same version, also where README/CHANGELOG ride in). The new zip + package_files POST runs after that (release_agent_email.yml:531), so by then versionExists === true and the whole !versionExists block is skipped — package_files is never written.

Downstream consequence: readPackageFiles returns null (storage.ts:517), so toIndexEntry's packageFiles && zip guard yields pkg = undefined (catalog.ts:382), and packageDownloadUrl / the {agent.package && pkgUrl && (...)} block render nothing. The zip artifact itself is still stored (the BUCKET.put at publish.ts:241 is unconditional), so the download exists at its URL — but the page that's supposed to surface it never does. Silent no-op, no error — the opposite of the "fail loudly" rule.

README/CHANGELOG are correctly gated to first-publish because they're sent on that first call. package_files is structurally different: it can only arrive on a later publish to an existing version, so it must not share that gate. Suggested fix — move the store out of the !versionExists block (a later binary publish passes no package_files, so packageFilesText is null there and won't clobber):

  if (!versionExists) {
    await env.BUCKET.put(rawManifestKey(manifest.id, manifest.version), manifestText, {
      httpMetadata: { contentType: "application/x-yaml; charset=utf-8" },
    });
    if (readmeText != null) {
      await env.BUCKET.put(readmeKey(manifest.id, manifest.version), readmeText, {
        httpMetadata: { contentType: "text/markdown; charset=utf-8" },
      });
    }
    if (changelogText != null) {
      await env.BUCKET.put(changelogKey(manifest.id, manifest.version), changelogText, {
        httpMetadata: { contentType: "text/markdown; charset=utf-8" },
      });
    }
  }
  // The whole-package file list rides with the .zip artifact, which is published
  // AFTER the binaries (i.e. to an already-existing version), so it must NOT be
  // gated on first-publish. A later binary publish sends no package_files, so it
  // won't clobber an existing listing.
  if (packageFilesText != null) {
    await env.BUCKET.put(packageFilesKey(manifest.id, manifest.version), packageFilesText, {
      httpMetadata: { contentType: "application/json; charset=utf-8" },
    });
  }

(If write-once immutability for the listing matters, additionally guard with a BUCKET.head(packageFilesKey(...)) check before the put — but don't keep it under !versionExists.)

🟡 Important

Test never exercises the two-publish ordering the workflow actually uses (workers/agent-hub/test/publish.test.ts:598)

All three new tests publish the zip + packageFiles in a single publish() call to a fresh env, so versionExists is always false — the exact state that hides the bug above. Add a test that mirrors the pipeline: publish a platform binary first, then publish the .zip + packageFiles as a second call to the same version, and assert entry.package is defined with the file list. That test fails today and will guard the fix.

it("surfaces package when the zip + file list are published AFTER the binaries (same version)", async () => {
  const env = makeEnv();
  await publish(env, {
    token: "tok_amd",
    manifestYaml: sampleManifest({ id: "two-step" }),
    artifact: "BINBYTES",
    filename: "email-agent-linux-x64",
  });
  await publish(env, {
    token: "tok_amd",
    manifestYaml: sampleManifest({ id: "two-step" }),
    artifact: "ZIPBYTES",
    filename: "agent-two-step-0.1.0.zip",
    packageFiles: filesJson,
  });
  const entry = (
    (await (await env.bucket.get("index.json"))!.json()) as CatalogIndex
  ).agents.find((a) => a.id === "two-step")!;
  expect(entry.package).toBeDefined();
  expect(entry.package!.files).toHaveLength(3);
});

🟢 Minor

size_bytes: 0 doc comment is now stale (workers/agent-hub/src/types.ts:216)PackageInfo.size_bytes says "0 if the artifact isn't found", but toIndexEntry only constructs pkg when a .zip artifact is found (catalog.ts:382), so the 0 case is unreachable. Drop the "0 if…" clause to avoid implying a fallback that doesn't exist.

Strengths

  • Worker/website/schema/README/types all updated together — the doc-sync discipline CLAUDE.md asks for is followed exactly.
  • gen_package_files.py fails loudly on an empty/missing staging dir rather than emitting an empty list, and optionalPackageFiles rejects a malformed part with an actionable 400 — both align with the no-silent-fallbacks rule.
  • The catalog only surfaces package when both the .zip artifact and its listing exist, and the website hides the whole section gracefully otherwise — good defensive pairing (the bug is purely that the listing never gets stored, not in this guard).

The whole-package file listing was stored only inside the !versionExists block,
but a real release publishes the per-platform binaries first (creating the
version) and the whole-package zip + package_files in a SEPARATE later POST. So
package_files arrived when versionExists was already true and was silently
dropped — the catalog never got `package`, and the download button + file list
never rendered in production. Every existing test did a single isolated POST, so
versionExists was false there: green tests, dead feature (a cold-state miss).

- Move the package_files write out of the first-POST gate; key it per version and
  guard with a head() check (the immutable zip artifact 409s before here anyway).
- Add a regression test that publishes a binary first, THEN the zip+package_files
  in a second POST, asserting index.json ends up with `package` populated. It
  fails on the old code and passes on the fix.
- Edge fetch-verify the published zip (200 + Content-Length == local size, same
  bounded retry as the binaries). The zip rides the artifact path and is SHA-
  verified on upload, but it is not in binaries.lock.json so the fetch CLI did not
  cover it — a non-propagated/edge-blocked zip would have shipped a 404 button.

worker: 64 tests (+1) pass, tsc clean.
@kovtcharov

Copy link
Copy Markdown
Contributor Author

Deep pre-release review found one release-blocking bug — now fixed in 2181d520. The byte-integrity story for the zip was already solid; the break was in when the file listing got stored.

Blocker: the file listing was never written on a real release — the whole feature was invisible in production. The Worker stored package-files.json only inside the first-POST (!versionExists) block. But a real release publishes the per-platform binaries first (which creates the version), then the whole-package zip + package_files in a separate later POST — so the listing arrived when versionExists was already true and got silently dropped. The catalog never got package, so the download button + file list never rendered. Every existing test did a single isolated POST (versionExists false), so they were green while the release path was dead — a cold-state miss.

🔍 Fix + the other items
  • publish.ts — moved the package_files write out of the !versionExists gate; keyed per version and guarded with a head() check (the immutable zip artifact 409s before reaching it anyway, so it can't be rewritten).
  • publish.test.ts — new regression test: publish a binary first, then the zip + package_files in a second POST, assert index.json ends up with package populated. Verified it fails on the old code (expected undefined to be defined) and passes on the fix. Suite 63 → 64.
  • release_agent_email.ymledge fetch-verify the zip (200 + Content-Length == local size, same bounded retry as the binaries). The zip is SHA-verified on upload, but it isn't in binaries.lock.json so the fetch-CLI loop didn't cover it — a non-propagated/edge-blocked zip would have shipped a 404 button.

Verified clean: zip byte integrity (server + local SHA, fail-loud), graceful degradation for agents without a zip (package? optional through types/schema/page), XSS-safe rendering, no platform-name skew. worker: 64 tests pass, tsc clean.

Merge-order note (with #1846): both PRs add a <section> to [id].astro at the same anchor → guaranteed conflict. Recommend merging this PR first, then rebasing #1846 keeping both cards. And this feature needs a manual wrangler deploy of the agent-hub Worker (no CI auto-deploy) before the zip surfaces live.

@kovtcharov-amd
kovtcharov-amd added this pull request to the merge queue Jun 24, 2026
Merged via the queue into main with commit f2d4eb1 Jun 24, 2026
25 checks passed
@kovtcharov-amd
kovtcharov-amd deleted the feat/hub-package-zip-download branch June 24, 2026 18:04
pull Bot pushed a commit to bhardwajRahul/gaia that referenced this pull request Jun 25, 2026
…ail 0.2.4 (amd#1855)

Gets the email-agent release unblocked. The ~177 MB all-platforms
whole-package zip (amd#1843) exceeds **Cloudflare's edge upload limit** —
`POST /publish` returns `413 Payload Too Large` before the request even
reaches the worker, which blocked the release on 0.2.1–0.2.3 (each
published its binaries but never the zip/npm). That edge limit is
plan-based and not fixable in worker code, so the worker-side streaming
attempt ([amd#1849](amd#1849)) couldn't have
solved it.

This reverts that streaming change and disables the whole-package zip
publish (`if: false`, with a note to revive it via presigned-to-R2 or
per-platform zips). **0.2.4 ships the per-platform binaries + npm client
+ website with no combined zip** — the binaries stay individually
downloadable from the Hub. No agent wire-contract change
(`SCHEMA_VERSION` stays 2.0).

<details>
<summary>What's in here</summary>

- **Revert amd#1849** — restores the worker + `publish_to_r2.py` to the
multipart-only path; worker suite back to 64 tests passing.
- **Disable the zip steps** — "Assemble + publish the whole-package zip"
and its edge-verify step are `if: false`; the binary fetch-verify, npm
publish, and website redeploy are untouched.
- **0.2.4 bump** — all six version targets synced via `stamp_version.py`
+ CHANGELOG.
</details>

## Test plan
- [x] `stamp_version.py --check` passes (all targets 0.2.4)
- [x] worker `npm run typecheck && npm test` — 64 pass (revert clean)
- [x] `cd hub/agents/npm/agent-email && npm run build && npm test` — 46
pass
- [ ] Tag `agent-pkg-email-v0.2.4` → release publishes binaries + npm
@0.2.4 + website (no zip step)

---------

Co-authored-by: Tomasz Iniewicz <tomasz@iniewicz.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

devops DevOps/infrastructure changes website GAIA website (amd-gaia.ai)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants